Method and system for outputting augmented reality information

ABSTRACT

A method and system are disclosed for outputting augmented reality information to a first user. In an embodiment, the method includes acquiring first information, including image information, depth information, coordinate information and combinations thereof, the first information relating to at least one of a medical device and a medical examination of a patient; creating the augmented reality information, relating to the medical device and/or the medical examination of the patient, based on the first information; and outputting the augmented reality information such that the augmented reality information is perceivable in a field of view of the first user.

PRIORITY STATEMENT

The present application is a continuation of and hereby claims thebenefit of priority under 35 U.S.C. § 120/121 to pending U.S.application Ser. No. 16/430,744 filed Jun. 4, 2019, which is acontinuation of and which claims the benefit of priority under 35 U.S.C.§ 120/121 to U.S. application Ser. No. 15/378,741 filed Dec. 14, 2016and now U.S. Pat. No. 10,366,489 issued Jul. 30, 2019, which claims thebenefit of priority under 35 U.S.C. § 119 to German patent applicationnumber DE 102015226669.9 filed Dec. 23, 2015, the entire contents ofeach of which are hereby incorporated herein by reference.

FIELD

At least one embodiment of the invention generally relates to a methodfor outputting augmented reality information, which relates to a medicaldevice and/or to a medical examination of a patient, to a first user. Atleast one embodiment of the invention further generally relates to asystem for outputting augmented reality information, to a computerprogram product and/or to a computer-readable medium.

BACKGROUND

A number of users with different specialist knowledge and perspectivesare typically involved both in using a medical device, for example amedical imaging device, and also in a medical examination of a patient.These especially include patients, medical technicians, physicians withvarying degrees of experience, technical staff and also makers ofmedical devices. Communication between these users is important for anefficient use of the medical device or for the success of the medicalexamination. In such cases there has previously been recourse to a greatextent to verbal communication without any technical support. Inaddition known communication aids such as brochures, information sheets,information films, operating instructions etc. are used.

The medical examination of the patient can for example include anintervention, a therapy, in particular a radiation therapy, medicalimaging, a clinical workflow, an instruction to the patient or similar.

In an intervention it is important to hit the target region preciselyand to protect the surrounding healthy tissue. Therefore theintervention will be supported in many cases via medical imaging.Previously the transmission of the imaging information provided by wayof the medical imaging to the intervention depended to a great extent onthe capabilities, the experience and the spatial sense of the physicianwho is carrying out the intervention. Typically the imaging informationprovided by way of the medical imaging is displayed on one or moremonitors. The monitor or the number of monitors is each arranged on aflexible support structure in an examination room in which theintervention is taking place. A relatively large distance between such amonitor and the intervention area can mean an enormous additional loadfor a physician, in particular when their ability to see the monitor isrestricted. In addition the physician must often shift their view, inparticular their focus, back and forth between the intervention area andthe one or more monitors during the course of the intervention. In manycases this is also associated with an ergonomic load on the physician.

In a clinical workflow, for example in radiology, it is important thatthe patient receives clear instructions at the right time in order topass through the various stations and steps efficiently. Furtherimportant aspects for a clinical workflow are the wellbeing of thepatient, in particular in relation to anxieties, uncertainties andrestrictions of the private sphere, as well as the protection ofpersonal data. With medical imaging too timely and clear instructions,for example instructions affecting breathing, as well as the wellbeingof the patient are important.

SUMMARY

At least one embodiment of the invention includes at least one of animproved use of a medical device and an improved execution of a medicalexamination of a patient.

At least one embodiment of the invention is directed to a method, asystem, a computer program product and/or a computer-readable medium.

In at least one embodiment of the inventive method for outputtingaugmented reality information, which relates to a medical device and/orto a medical examination of a patient, to a first user, firstinformation is acquired, which is selected from the group that consistsof image information, depth information, coordinate information andcombinations thereof, wherein the first information relates to themedical device and/or to the medical examination. Based on the firstinformation the augmented reality information, which relates to themedical device and/or the medical examination of the patient, iscreated. The augmented reality information is output such that theaugmented reality information is able to be perceived in a field of viewof the first user.

In accordance with one embodiment of the invention a movement parameter,which relates to a movement of a movable component of the medicaldevice, can be acquired. Preferably the coordinate information isestablished based on the movement parameter. The movement parameter canin particular depend on a movement of the movable component. Inparticular the movement parameter can be a measure for a movement of themovable component. The movement parameter can for example specify ageometrical variable, in particular a location, a length and/or anangle. The movement parameter can for example specify an electricvariable, in particular a current, a voltage, a charge and/or an energy.The electric variable can for example relate to a drive unit with whichthe movement of the movable component is driven.

In accordance with one embodiment of the invention, a medical dataset isprovided by way of the medical device, wherein the augmented realityinformation is created based on the medical dataset. In accordance withone form of embodiment of the invention the medical dataset features amedical imaging dataset. In particular the medical imaging dataset canrelate to a structure of the patient. In accordance with a further formof embodiment of the invention the medical dataset features a medicalsignal, which relates to the patient. The medical signal can for examplebe a physiological signal and/or a bioelectric signal. In particular themedical signal can relate to a heartbeat of the patient, breathing ofthe patient, a body temperature of the patient, a concentration of asubstance in the blood of the patient or similar. The medical signal canfor example be an electrocardiogram signal (EKG signal).

At least one embodiment of the inventive system for outputting augmentedreality information, which relates to a medical device and/or to amedical examination of a patient, to a first user, features a firstacquisition module, a creation module and a first output module. Thefirst acquisition module is embodied for acquiring first informationthat is selected from the group that consists of image information,depth information, coordinate information and combinations thereof,wherein the first information relates to the medical device and/or themedical examination. The creation module is embodied for creating theaugmented reality information that relates to the medical device and/orthe medical examination of the patient, based on the first information.The first output module is embodied for outputting augmented realityinformation such that the augmented reality information is able to beperceived in a field of view of the first user.

One form of embodiment of the invention makes provision for theinventive system and/or for a number of components of the inventivesystem to be realized at least partly in the form of software on aprocessor system. In particular the first acquisition module, thecreation module, the first output module, the information provisionmodule, the second output module and the second acquisition module caneach form a component of the inventive system and/or each be realized atleast partly in the form of software on a processor system. One form ofembodiment of the invention makes provision for the inventive systemand/or a or a number of components of the inventive system to berealized at least partly in the form of software-supported hardware, forexample FPGAs, a processor system or the like.

One form of embodiment of the invention makes provision for theinventive system and/or for a number of components of the inventivesystem to be formed at least partly by a Cloud via Cloud Computing. Oneform of embodiment of the invention makes provision for the creationmodule and/or at least a submodule of the creation modules to be formedby a Cloud via Cloud Computing. The Cloud can in particular feature anetwork of memory areas spatially remote from one another and processorsystems spatially remote from one another. For data transfer from theCloud and/or to the Cloud the first augmented reality device can featurea first Cloud interface and/or the second augmented reality device canfeature a second Cloud interface.

The described method and the described system merely involve forms ofembodiment of the invention. The invention can be varied by the personskilled in the art without departing from the area of the invention,provided it is specified by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be explained once again in greater detail below withreference to the enclosed figures based on example embodiments. Therepresentation in the figures is schematic and greatly simplified andalso not absolutely true-to-scale.

In the figures:

FIG. 1 shows a flow diagram of a method in accordance with a first formof embodiment of the invention,

FIG. 2 shows a diagram of a system in accordance with a second form ofembodiment of the invention,

FIG. 3 shows a system in accordance with a third form of embodiment ofthe invention,

FIG. 4 shows a flow diagram of a method in accordance with a fourth formof embodiment of the invention,

FIG. 5 shows a diagram of a system in accordance with a fifth form ofembodiment of the invention.

FIG. 6 shows a system in accordance with a sixth form of embodiment ofthe invention.

FIG. 7 shows a section from the field of view of the second user,wherein the field of view is enhanced with the augmented realityinformation.

DETAILED DESCRIPTION OF THE EXAMPLE EMBODIMENTS

The drawings are to be regarded as being schematic representations andelements illustrated in the drawings are not necessarily shown to scale.Rather, the various elements are represented such that their functionand general purpose become apparent to a person skilled in the art. Anyconnection or coupling between functional blocks, devices, components,or other physical or functional units shown in the drawings or describedherein may also be implemented by an indirect connection or coupling. Acoupling between components may also be established over a wirelessconnection. Functional blocks may be implemented in hardware, firmware,software, or a combination thereof.

Various example embodiments will now be described more fully withreference to the accompanying drawings in which only some exampleembodiments are shown. Specific structural and functional detailsdisclosed herein are merely representative for purposes of describingexample embodiments. Example embodiments, however, may be embodied invarious different forms, and should not be construed as being limited toonly the illustrated embodiments. Rather, the illustrated embodimentsare provided as examples so that this disclosure will be thorough andcomplete, and will fully convey the concepts of this disclosure to thoseskilled in the art. Accordingly, known processes, elements, andtechniques, may not be described with respect to some exampleembodiments. Unless otherwise noted, like reference characters denotelike elements throughout the attached drawings and written description,and thus descriptions will not be repeated. The present invention,however, may be embodied in many alternate forms and should not beconstrued as limited to only the example embodiments set forth herein.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, components, regions,layers, and/or sections, these elements, components, regions, layers,and/or sections, should not be limited by these terms. These terms areonly used to distinguish one element from another. For example, a firstelement could be termed a second element, and, similarly, a secondelement could be termed a first element, without departing from thescope of example embodiments of the present invention. As used herein,the term “and/or,” includes any and all combinations of one or more ofthe associated listed items. The phrase “at least one of” has the samemeaning as “and/or”.

Spatially relative terms, such as “beneath,” “below,” “lower,” “under,”“above,” “upper,” and the like, may be used herein for ease ofdescription to describe one element or feature's relationship to anotherelement(s) or feature(s) as illustrated in the figures. It will beunderstood that the spatially relative terms are intended to encompassdifferent orientations of the device in use or operation in addition tothe orientation depicted in the figures. For example, if the device inthe figures is turned over, elements described as “below,” “beneath,” or“under,” other elements or features would then be oriented “above” theother elements or features. Thus, the example terms “below” and “under”may encompass both an orientation of above and below. The device may beotherwise oriented (rotated 90 degrees or at other orientations) and thespatially relative descriptors used herein interpreted accordingly. Inaddition, when an element is referred to as being “between” twoelements, the element may be the only element between the two elements,or one or more other intervening elements may be present.

Spatial and functional relationships between elements (for example,between modules) are described using various terms, including“connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitlydescribed as being “direct,” when a relationship between first andsecond elements is described in the above disclosure, that relationshipencompasses a direct relationship where no other intervening elementsare present between the first and second elements, and also an indirectrelationship where one or more intervening elements are present (eitherspatially or functionally) between the first and second elements. Incontrast, when an element is referred to as being “directly” connected,engaged, interfaced, or coupled to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between,” versus “directly between,” “adjacent,” versus“directly adjacent,” etc.).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of exampleembodiments of the invention. As used herein, the singular forms “a,”“an,” and “the,” are intended to include the plural forms as well,unless the context clearly indicates otherwise. As used herein, theterms “and/or” and “at least one of” include any and all combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “comprises,” “comprising,” “includes,” and/or“including,” when used herein, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items. Expressionssuch as “at least one of,” when preceding a list of elements, modify theentire list of elements and do not modify the individual elements of thelist. Also, the term “example” is intended to refer to an example orillustration.

When an element is referred to as being “on,” “connected to,” “coupledto,” or “adjacent to,” another element, the element may be directly on,connected to, coupled to, or adjacent to, the other element, or one ormore other intervening elements may be present. In contrast, when anelement is referred to as being “directly on,” “directly connected to,”“directly coupled to,” or “immediately adjacent to,” another elementthere are no intervening elements present.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which example embodiments belong. Itwill be further understood that terms, e.g., those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Before discussing example embodiments in more detail, it is noted thatsome example embodiments may be described with reference to acts andsymbolic representations of operations (e.g., in the form of flowcharts, flow diagrams, data flow diagrams, structure diagrams, blockdiagrams, etc.) that may be implemented in conjunction with units and/ordevices discussed in more detail below. Although discussed in aparticularly manner, a function or operation specified in a specificblock may be performed differently from the flow specified in aflowchart, flow diagram, etc. For example, functions or operationsillustrated as being performed serially in two consecutive blocks mayactually be performed simultaneously, or in some cases be performed inreverse order. Although the flowcharts describe the operations assequential processes, many of the operations may be performed inparallel, concurrently or simultaneously. In addition, the order ofoperations may be re-arranged. The processes may be terminated whentheir operations are completed, but may also have additional steps notincluded in the figure. The processes may correspond to methods,functions, procedures, subroutines, subprograms, etc.

Specific structural and functional details disclosed herein are merelyrepresentative for purposes of describing example embodiments of thepresent invention. This invention may, however, be embodied in manyalternate forms and should not be construed as limited to only theembodiments set forth herein.

Units and/or devices according to one or more example embodiments may beimplemented using hardware, software, and/or a combination thereof. Forexample, hardware devices may be implemented using processing circuitrysuch as, but not limited to, a processor, Central Processing Unit (CPU),a controller, an arithmetic logic unit (ALU), a digital signalprocessor, a microcomputer, a field programmable gate array (FPGA), aSystem-on-Chip (SoC), a programmable logic unit, a microprocessor, orany other device capable of responding to and executing instructions ina defined manner. Portions of the example embodiments and correspondingdetailed description may be presented in terms of software, oralgorithms and symbolic representations of operation on data bits withina computer memory. These descriptions and representations are the onesby which those of ordinary skill in the art effectively convey thesubstance of their work to others of ordinary skill in the art. Analgorithm, as the term is used here, and as it is used generally, isconceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of optical, electrical, or magnetic signals capable of beingstored, transferred, combined, compared, and otherwise manipulated. Ithas proven convenient at times, principally for reasons of common usage,to refer to these signals as bits, values, elements, symbols,characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, or as is apparent from the discussion,terms such as “processing” or “computing” or “calculating” or“determining” of “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computingdevice/hardware, that manipulates and transforms data represented asphysical, electronic quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

In this application, including the definitions below, the term ‘module’or the term ‘controller’ may be replaced with the term ‘circuit.’ Theterm ‘module’ may refer to, be part of, or include processor hardware(shared, dedicated, or group) that executes code and memory hardware(shared, dedicated, or group) that stores code executed by the processorhardware.

The module may include one or more interface circuits. In some examples,the interface circuits may include wired or wireless interfaces that areconnected to a local area network (LAN), the Internet, a wide areanetwork (WAN), or combinations thereof. The functionality of any givenmodule of the present disclosure may be distributed among multiplemodules that are connected via interface circuits. For example, multiplemodules may allow load balancing. In a further example, a server (alsoknown as remote, or cloud) module may accomplish some functionality onbehalf of a client module.

Software may include a computer program, program code, instructions, orsome combination thereof, for independently or collectively instructingor configuring a hardware device to operate as desired. The computerprogram and/or program code may include program or computer-readableinstructions, software components, software modules, data files, datastructures, and/or the like, capable of being implemented by one or morehardware devices, such as one or more of the hardware devices mentionedabove. Examples of program code include both machine code produced by acompiler and higher level program code that is executed using aninterpreter.

For example, when a hardware device is a computer processing device(e.g., a processor, Central Processing Unit (CPU), a controller, anarithmetic logic unit (ALU), a digital signal processor, amicrocomputer, a microprocessor, etc.), the computer processing devicemay be configured to carry out program code by performing arithmetical,logical, and input/output operations, according to the program code.Once the program code is loaded into a computer processing device, thecomputer processing device may be programmed to perform the programcode, thereby transforming the computer processing device into a specialpurpose computer processing device. In a more specific example, when theprogram code is loaded into a processor, the processor becomesprogrammed to perform the program code and operations correspondingthereto, thereby transforming the processor into a special purposeprocessor.

Software and/or data may be embodied permanently or temporarily in anytype of machine, component, physical or virtual equipment, or computerstorage medium or device, capable of providing instructions or data to,or being interpreted by, a hardware device. The software also may bedistributed over network coupled computer systems so that the softwareis stored and executed in a distributed fashion. In particular, forexample, software and data may be stored by one or more computerreadable recording mediums, including the tangible or non-transitorycomputer-readable storage media discussed herein.

Even further, any of the disclosed methods may be embodied in the formof a program or software. The program or software may be stored on anon-transitory computer readable medium and is adapted to perform anyone of the aforementioned methods when run on a computer device (adevice including a processor). Thus, the non-transitory, tangiblecomputer readable medium, is adapted to store information and is adaptedto interact with a data processing facility or computer device toexecute the program of any of the above mentioned embodiments and/or toperform the method of any of the above mentioned embodiments.

Example embodiments may be described with reference to acts and symbolicrepresentations of operations (e.g., in the form of flow charts, flowdiagrams, data flow diagrams, structure diagrams, block diagrams, etc.)that may be implemented in conjunction with units and/or devicesdiscussed in more detail below. Although discussed in a particularlymanner, a function or operation specified in a specific block may beperformed differently from the flow specified in a flowchart, flowdiagram, etc. For example, functions or operations illustrated as beingperformed serially in two consecutive blocks may actually be performedsimultaneously, or in some cases be performed in reverse order.

According to one or more example embodiments, computer processingdevices may be described as including various functional units thatperform various operations and/or functions to increase the clarity ofthe description. However, computer processing devices are not intendedto be limited to these functional units. For example, in one or moreexample embodiments, the various operations and/or functions of thefunctional units may be performed by other ones of the functional units.Further, the computer processing devices may perform the operationsand/or functions of the various functional units without sub-dividingthe operations and/or functions of the computer processing units intothese various functional units.

Units and/or devices according to one or more example embodiments mayalso include one or more storage devices. The one or more storagedevices may be tangible or non-transitory computer-readable storagemedia, such as random access memory (RAM), read only memory (ROM), apermanent mass storage device (such as a disk drive), solid state (e.g.,NAND flash) device, and/or any other like data storage mechanism capableof storing and recording data. The one or more storage devices may beconfigured to store computer programs, program code, instructions, orsome combination thereof, for one or more operating systems and/or forimplementing the example embodiments described herein. The computerprograms, program code, instructions, or some combination thereof, mayalso be loaded from a separate computer readable storage medium into theone or more storage devices and/or one or more computer processingdevices using a drive mechanism. Such separate computer readable storagemedium may include a Universal Serial Bus (USB) flash drive, a memorystick, a Blu-ray/DVD/CD-ROM drive, a memory card, and/or other likecomputer readable storage media. The computer programs, program code,instructions, or some combination thereof, may be loaded into the one ormore storage devices and/or the one or more computer processing devicesfrom a remote data storage device via a network interface, rather thanvia a local computer readable storage medium. Additionally, the computerprograms, program code, instructions, or some combination thereof, maybe loaded into the one or more storage devices and/or the one or moreprocessors from a remote computing system that is configured to transferand/or distribute the computer programs, program code, instructions, orsome combination thereof, over a network. The remote computing systemmay transfer and/or distribute the computer programs, program code,instructions, or some combination thereof, via a wired interface, an airinterface, and/or any other like medium.

The one or more hardware devices, the one or more storage devices,and/or the computer programs, program code, instructions, or somecombination thereof, may be specially designed and constructed for thepurposes of the example embodiments, or they may be known devices thatare altered and/or modified for the purposes of example embodiments.

A hardware device, such as a computer processing device, may run anoperating system (OS) and one or more software applications that run onthe OS. The computer processing device also may access, store,manipulate, process, and create data in response to execution of thesoftware. For simplicity, one or more example embodiments may beexemplified as a computer processing device or processor; however, oneskilled in the art will appreciate that a hardware device may includemultiple processing elements or processors and multiple types ofprocessing elements or processors. For example, a hardware device mayinclude multiple processors or a processor and a controller. Inaddition, other processing configurations are possible, such as parallelprocessors.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium (memory).The computer programs may also include or rely on stored data. Thecomputer programs may encompass a basic input/output system (BIOS) thatinteracts with hardware of the special purpose computer, device driversthat interact with particular devices of the special purpose computer,one or more operating systems, user applications, background services,background applications, etc. As such, the one or more processors may beconfigured to execute the processor executable instructions.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language) or XML (extensible markuplanguage), (ii) assembly code, (iii) object code generated from sourcecode by a compiler, (iv) source code for execution by an interpreter,(v) source code for compilation and execution by a just-in-timecompiler, etc. As examples only, source code may be written using syntaxfrom languages including C, C++, C#, Objective-C, Haskell, Go, SQL, R,Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5,Ada, ASP (active server pages), PHP, Scala, Eiffel, Smalltalk, Erlang,Ruby, Flash®, Visual Basic®, Lua, and Python®.

Further, at least one embodiment of the invention relates to thenon-transitory computer-readable storage medium including electronicallyreadable control information (procesor executable instructions) storedthereon, configured in such that when the storage medium is used in acontroller of a device, at least one embodiment of the method may becarried out.

The computer readable medium or storage medium may be a built-in mediuminstalled inside a computer device main body or a removable mediumarranged so that it can be separated from the computer device main body.The term computer-readable medium, as used herein, does not encompasstransitory electrical or electromagnetic signals propagating through amedium (such as on a carrier wave); the term computer-readable medium istherefore considered tangible and non-transitory. Non-limiting examplesof the non-transitory computer-readable medium include, but are notlimited to, rewriteable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewriteable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. Shared processor hardware encompasses asingle microprocessor that executes some or all code from multiplemodules. Group processor hardware encompasses a microprocessor that, incombination with additional microprocessors, executes some or all codefrom one or more modules. References to multiple microprocessorsencompass multiple microprocessors on discrete dies, multiplemicroprocessors on a single die, multiple cores of a singlemicroprocessor, multiple threads of a single microprocessor, or acombination of the above.

Shared memory hardware encompasses a single memory device that storessome or all code from multiple modules. Group memory hardwareencompasses a memory device that, in combination with other memorydevices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium is therefore considered tangible and non-transitory. Non-limitingexamples of the non-transitory computer-readable medium include, but arenot limited to, rewriteable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewriteable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks andflowchart elements described above serve as software specifications,which can be translated into the computer programs by the routine workof a skilled technician or programmer.

Although described with reference to specific examples and drawings,modifications, additions and substitutions of example embodiments may bevariously made according to the description by those of ordinary skillin the art. For example, the described techniques may be performed in anorder different with that of the methods described, and/or componentssuch as the described system, architecture, devices, circuit, and thelike, may be connected or combined to be different from theabove-described methods, or results may be appropriately achieved byother components or equivalents.

In at least one embodiment of the inventive method for outputtingaugmented reality information, which relates to a medical device and/orto a medical examination of a patient, to a first user, firstinformation is acquired, which is selected from the group that consistsof image information, depth information, coordinate information andcombinations thereof, wherein the first information relates to themedical device and/or to the medical examination. Based on the firstinformation the augmented reality information, which relates to themedical device and/or the medical examination of the patient, iscreated. The augmented reality information is output such that theaugmented reality information is able to be perceived in a field of viewof the first user.

Augmented reality information is especially to be understood asinformation with which a reality, in particular the field of view of thefirst user and/or a field of view of a second user, can be augmented. Inparticular an augmented reality can be created on the basis of the fieldof view of the first user and/or on the basis of the field of view ofthe second user with the augmented reality information. The augmentedreality information is in particular able to be perceived in the fieldof view of the first user when the augmented reality information isvisible for the first user in the field of view of the first user. Inparticular the augmented reality information can be output such that thefield of view of the first user is augmented by the augmented realityinformation.

The augmented reality information can be created and/or output forexample via an augmented reality device. The term “device for creatingan augmented reality” and the term “augmented reality device” are usedsynonymously. An enhancement of a reality is known in particular to theperson skilled in the art as “augmentation”. An enhanced reality isknown in particular to the person skilled in the art as “augmentedreality”. An enhanced reality device is known in particular to theperson skilled in the art as an “augmented reality device”.

A known device for creating an augmented reality is e.g. the MicrosoftHoloLens® made by Microsoft Corporation, USA. The structure and thefunctioning of a device for creating an augmented reality are known tothe person skilled in the art, in particular from US2014098226A1,US2015248793A1, US2015294507A1, US2015234455A1, WO2015134958A1,WO2015161307A1 and [06], the entire contents of each of which are herebyincorporated herein by reference.

One embodiment of the invention makes provision for the described methodand/or one or more steps of the described method to each be carried outautomatically or fully automatically. In particular the augmentedreality information can be created automatically and/or fullyautomatically, in particular via the augmented reality device.“Automatically” in the context of the present application means that therespective step is carried out autonomously via software and/or viahardware and/or that essentially no interaction of a user is necessaryfor the respective step. Essentially no interaction of a user isnecessary, particularly when one or more results are merely to beconfirmed and/or one or more intermediate steps are merely to be carriedout by a user. “Fully automatically” in the context of the presentapplication means that no interaction at all of a user is needed for theexecution of the respective step. Regardless of whether one or moresteps are carried out “automatically” or “fully automatically”, theinventive method can be an element of a workflow that additionallyrequires an interaction of a user. The interaction of the user can forexample consist of the user manually creating or selecting anexamination protocol and/or an examination plan and/or a clinicalproblem, from a menu presented via a screen for example.

As an alternative or in addition, the augmented reality information canbe created on the basis of an input of the first user and/or based on aninput of a second user. For example, based on an input of the first userand/or based on an input of the second user, a marking can be createdand/or adapted in relation to a type, a position, a size and/or a shape,wherein the augmented reality information features the marking.

The image information can for example be a camera image and/or a 2Dimage (two-dimensional image). One form of embodiment of the inventionmakes provision for the first information to be a 3D image(three-dimensional image). The 3D image can in particular be acombination of a 2D image and depth information. In this case each pixelof the 3D image can be assigned an image value of the 2D image and adepth value of the depth information.

The image information can be acquired for example via a first cameraunit. The first camera unit can be a 3D camera for example. The 3D imageand/or the depth information can be acquired for example via the 3Dcamera. The first augmented reality device can for example feature thefirst camera unit. The 3D camera can be designed in particular fordetection of electromagnetic radiation, in particular for detection ofelectromagnetic radiation in a lower-frequency range of the spectrumcompared to x-ray radiation, for example in the visible or infraredrange of the spectrum. The 3D camera can be embodied for example as astereo camera or as a time-of-flight measurement system. Atime-of-flight measurement system is known in particular to the personskilled in the art as a “time-of-flight camera”. The 3D camera can beembodied for example for acquiring the 3D image and/or for acquiring thedepth information by way of structured illumination.

The coordinate information can relate for example to a map dataset. Themap dataset can for example feature a map of a spatial area in a fieldof view of the first user. In particular the at least one part of themedical device can be located in the spatial area and/or at least onepart step of the medical examination can take place in the spatial area.The coordinate information can for example specify a position and/or anorientation in particular in relation to the map dataset and/or inrelation to the map of the spatial area.

In accordance with one embodiment of the invention, the coordinateinformation comprises at least one patient coordinate, which relates toat least one part of the patient, and/or at least one device coordinate,which relates to at least one part of the medical device. The at leastone patient coordinate can for example specify a position and/or anorientation of the at least one part of the patient, in particular inrelation to the map dataset and/or in relation to the map of the spatialarea. The at least one device coordinate can for example specify aposition and/or an orientation of the at least one part of the medicaldevice, in particular in relation to the map dataset and/or in relationto the map of the spatial area. The at least one part of the medicaldevice can for example be a movable component of the medical device.

One form of embodiment of the invention makes provision for a positionand/or a direction of view of the first user to be determined inparticular in relation to the map dataset and/or in relation to the mapof the spatial area and/or for a position and/or an alignment of thefirst augmented reality device to be determined in particular inrelation to the map dataset and/or in relation to the map of the spatialarea. The augmented reality information can be created for example basedon the position and/or the direction of view of the first user and/orbased on the position and/or the alignment of the first augmentedreality device.

This can be done for example via the image information, via the depthinformation and/or via movement information. The movement informationcan be acquired for example via one or more acceleration sensors. Inparticular the first augmented reality device can feature one or moreacceleration sensors.

In a medical device, which has a patient support device with a transferplate, the coordinate information can specify the position and/or theorientation of the transfer plate for example. In a medical device,which has a kinematic chain and/or a robot arm, in particular a C-arm,the coordinate information can specify the position and/or theorientation in each case for one or more elements of the kinematic chainand/or of the robot arm for example.

A plurality of different representations is conceivable for thecoordinate information. There can be selection and/or transformationbetween these representations, without departing from the area of theinvention, provided it is specified by the claims.

For example coordinate information that relates to a transfer plate, ina first representation can feature coordinates of four corner points ofthe transfer plate, in a second representation coordinates of a centralpoint of the transfer plate as well as coordinates that specify theorientation of a plane approximately corresponding to the transferplate, and in a third representation can feature one coordinate value,wherein the coordinate value is assigned a position and/or anorientation of the transfer plate, for example via a list, in particulara lookup table.

The coordinate information can for example be established by a positionand/or an orientation being established, in particular measured. Thecoordinate information can for example be established by a positionchange and/or an orientation change being established, in particularmeasured.

In accordance with one embodiment of the invention the coordinateinformation is provided via the medical device.

The coordinate information can for example relate to a movable componentof the medical device.

In accordance with one form of embodiment of the invention a devicecoordinate system is provided for the medical device. In this case themovable component can be controlled on the basis of the devicecoordinate system, for example via a control device of the medicaldevice, in particular positioned and/or oriented. The coordinateinformation can for example comprise a device coordinate in relation tothe device coordinate system.

One form of embodiment of the invention makes provision, during thecreation of the augmented reality information, for a transformation ofthe coordinate information and/or of one of the coordinates included inthe coordinate information, for example the device coordinate, to becarried out from a first reference system, for example the devicecoordinate system, into a second reference system, for example the mapdataset. The transformation can be carried out for example based ontransformation information that relates to the first reference systemand the second reference system and/or with which coordinates of thesecond reference system can be assigned coordinates of the firstreference system. For example the coordinate information can include thetransformation information. Optionally the transformation informationcan be established and/or acquired.

With the aid of the coordinate information the augmented realityinformation can be created independently of the image information and/orindependently of the depth information for example. In this case inparticular a device coordinate system already provided for control ofthe medical device can advantageously additionally be used for creatingthe augmented reality information. In particular with the aid of thecoordinate information in conjunction with a map dataset and/or a 3Dmodel of the movable component and/or of the medical device, a positionand/or an orientation of the movable component can be established veryexactly and independently of visual circumstances. Optionally thecoordinate information can be combined with the image information and/orwith the depth information.

In accordance with one embodiment of the invention a control command,which relates to a movement of a movable component of the medicaldevice, is output. Preferably the coordinate information is establishedbased on the control command. A movement of the movable component can becontrolled by way of the control command for example. In particular thecontrol command can specify a target position, a position change, atarget orientation, an orientation change and/or similar for the movablecomponent. In particular coordinate information can be assigned to thecontrol command and/or to the movement parameter.

In accordance with one embodiment of the invention a movement parameter,which relates to a movement of a movable component of the medicaldevice, can be acquired. Preferably the coordinate information isestablished based on the movement parameter. The movement parameter canin particular depend on a movement of the movable component. Inparticular the movement parameter can be a measure for a movement of themovable component. The movement parameter can for example specify ageometrical variable, in particular a location, a length and/or anangle. The movement parameter can for example specify an electricvariable, in particular a current, a voltage, a charge and/or an energy.The electric variable can for example relate to a drive unit with whichthe movement of the movable component is driven.

The movable component can for example be a C-arm, a robot arm, akinematic chain, a transfer plate, an intervention tool or similar.

In accordance with one form of embodiment of the invention, the firstinformation relates to a spatial area in a field of view of the firstuser. At least a part of the medical device can be located in thespatial area. At least a part step of the medical examination can takeplace in the spatial area. One form of embodiment of the invention makesprovision for at least a part of the medical device to be located in afield of view of the first user and/or for at least a part step of themedical examination to take place in the field of view of the firstuser.

A part step of the medical examination can for example be a part step ofan intervention and/or a part step of a clinical workflow. A part stepof an intervention can for example be an introduction of an interventiontool and/or of an implant into the patient, a separation and/or ajoining of a structure of the patient or similar. A part step of aclinical workflow can for example be calling a patient waiting in awaiting room, identifying the patient, clarify further steps of theclinical workflow for the patient, preparation of the patient in anexamination room, instruction of the patient, preparation of the medicaldevice, registration of the patient at the medical device, medicalimaging, an intervention, a therapy, a diagnosis or similar.

In accordance with one embodiment of the invention, a medical dataset isprovided by way of the medical device, wherein the augmented realityinformation is created based on the medical dataset. In accordance withone form of embodiment of the invention the medical dataset features amedical imaging dataset. In particular the medical imaging dataset canrelate to a structure of the patient. In accordance with a further formof embodiment of the invention the medical dataset features a medicalsignal, which relates to the patient. The medical signal can for examplebe a physiological signal and/or a bioelectric signal. In particular themedical signal can relate to a heartbeat of the patient, breathing ofthe patient, a body temperature of the patient, a concentration of asubstance in the blood of the patient or similar. The medical signal canfor example be an electrocardiogram signal (EKG signal).

One form of embodiment of the invention makes provision for theaugmented reality information to have an output field, wherein themedical dataset is represented in the output field. The medical datasetcan be represented in the output field for example as a medical image,as a timing characteristic, as a signal curve and/or as a number.

One form of embodiment of the invention makes provision for a positionof the medical device to be established relative to the field of view ofthe first user based on the first information, wherein the augmentedreality information is created based on the position of the medicaldevice. In this case, based on the position of the medical device, theposition of the output field relative to the field of view of the firstuser can be determined. The output field can be displayed for example inthe vicinity of the medical device and/or overlaid on the medicaldevice. The output field can follow the medical device for example for achange of the position of the medical device relative to the field ofview of the first user.

In accordance with one embodiment of the invention, examinationinformation, which relates to the medical examination of the patient, isestablished based on the first information, wherein the augmentedreality information is created based on the examination information.

The examination information can for example be information about whichpart steps of the medical examination have already been carried outand/or which part step is to be carried out as the next step. Inparticular the examination information can be established based on anexamination plan, which comprises planned part steps of the medicalexamination, and/or based on an examination protocol. An examinationplan can for example be an intervention plan and/or a flow plan of aclinical workflow. The examination information can for example involve aposition of the patient relative to the medical device and/or a positionof an intervention tool relative to a structure of the patient. Forexample the examination information can feature the information that anintervention tool is located in the correct position ready for use. Theaugmented reality information can feature a corresponding marking, whichrepresents the next movement of the intervention tool to be carried out,for example in the form of a simulation.

In accordance with one embodiment of the invention patient information,which relates to the patient, is established based on the firstinformation, wherein the augmented reality information is created basedon the patient information.

In accordance with one embodiment of the invention the patientinformation is selected from the group that consists of

-   -   Patient identification information, which relates to an        identification of the patient,    -   Patient state information, which relates to an physiological        state and/or to an emotional state of the patient,    -   Patient position information, which relates to a position of the        patient,        and combinations thereof.

The patient identification information is designed to identify thepatient. The identification can in particular be carried outautomatically and/or fully automatically. Based on the patientidentification information for example, patient data of the patient canbe selected from a patient database and/or output by way of theaugmented reality information. The patient data can for example comprisepersonal data and/or data that relates to the patient history of thepatient. In particular a check can be made in this way as to whether agiven part step of the medical examination and/or which part step of themedical examination is planned for the patient. In particular thepatient data and/or a result of the check can be output by way of theaugmented reality information. Based on the patient identificationinformation for example, data that will be acquired during the medicalexamination of the patient and/or by way of the medical device isautomatically assigned to the patient and/or stored in a patientdatabase. This thus enables incorrect examinations and incorrectassignments to be avoided.

The patient identification information can for example be establishedbased on the image information, in that biometric features of thepatient are established and/or in that a code, in particular a QR code,which is assigned to the patient and/or to a patient file of thepatient, is acquired and evaluated.

The patient status information can for example relate to breathing ofthe patient, a body temperature of the patient, a pulse of the patient,a pupil dilation of the patient, a facial expression of the patient orsimilar. Based on the patient state information it can be establishedfor example whether and/or to what degree the patient is exhibitingstress. A warning signal can be output by way of augmented realityinformation for example if the stress of the patient could adverselyaffect the success of the medical examination. The augmented realityinformation can for example feature a calming image and/or a sequence ofcalming images for calming the patient.

The position of the patient can for example be selected from the patientposition group, which consists of a position of the patient in a clinic,a position of the patient in an examination room, a position of thepatient relative to the medical device, a position of the patientrelative to the first user, a position of the patient relative to thesecond user and combinations thereof. The augmented reality informationcan for example be created based on the patient position informationsuch that the position of the augmented reality information relative tothe field of view of the first user is determined based on the positionof the patient. The augmented reality information can for example beshown in the vicinity of the patient and/or overlaid on the patient. Theaugmented reality information can for example follow the patient for achange of position of the patient relative to the field of view of thefirst user.

In accordance with one embodiment of the invention a medical imagingdataset, which relates to a structure of the patient, is provided,wherein the augmented reality information is created based on themedical imaging dataset. In accordance with one aspect of the inventionthe augmented reality information features a structure marking thatmarks the structure of the patient. The medical imaging dataset can beprovided for example by way of the medical device, in particular by wayof a medical imaging device, and/or by way of a medical image database.In particular the medical image database can feature a memory area inwhich the medical image dataset is held for retrieval.

The structure of the patient can for example be an anatomical structureof the patient. The structure of the patient can for example feature anorgan of the patient and/or an implant of the patient. The implant canfor example be a screw, a prosthesis, a vessel prosthesis, a heart valveprosthesis or similar. The structure marking can for example feature amedical image of the structure of the patient and/or be shown overlaidin the field of view of the first user on the structure of the patient.The medical image of the structure can be segmented into part structuresfor example. In such cases the structure of the patient does notnecessarily have to be visible. For example based on the firstinformation and/or based on further information, for example a topogramand/or anatomical landmarks, a position of the structure relative to thepatient and/or a position of the structure marking relative to the fieldof view of the first user, which corresponds to the position of thestructure relative to the patient, is established. The structure markingcan for example feature a 3D image of the structure and/or a 3D model ofthe structure. With the aid of the structure marking for example themedical examination, in particular an intervention, can be plannedand/or supported. The structure marking can for example feature athree-dimensional rendered image of the structure and/or an unfoldingrepresentation of the structure.

For improved visualization of medical problems or circumstances, acomputer-generated image can be overlaid on the patient in the correctlocation. In this way an in particular three-dimensionally renderedimage of a structure, in particular of an inner anatomy, of the patientcan be overlaid on the patient for example. With such an image forexample a path of an interventional procedure and/or for one or moresections of a vessel a diagnostic value in each case, e.g. a fractionalflow reserve (FFR), can be displayed. For example by way of theaugmented reality information an unfolding representation of the ribcage of the patient can be shown overlaid on the rib cage of thepatient.

One form of embodiment of the invention makes provision for anatomicallandmarks based on the first information and/or based on the medicalimaging dataset to be detected and/or used as orientation points. Basedon the anatomical landmarks the rendered image can be registered inrelation to the patient and/or in relation to the structure of thepatient. The rendered image is thus able to be shown overlaid onstructure of the patient.

In accordance with one embodiment of the invention nominal positioninformation, which relates to a nominal position of the patient and/or anominal position of the medical device, is provided, wherein theaugmented reality information is created based on the nominal positioninformation, wherein the augmented reality information features anominal position marking, which marks the nominal position of thepatient and/or the nominal position of the medical device.

The nominal position information then relates in particular to a nominalposition of the medical device, if the nominal position informationrelates to a nominal position of the movable component of the medicaldevice. The nominal position information can for example be establishedbased on a pre-examination, based on an examination plan and/or based onan examination protocol of the medical examination. The examinationprotocol can in particular be an imaging protocol. The nominal positioninformation can for example specify how the patient is to be positionedfor the medical examination and/or relative to the medical device. Thenominal position information can for example specify how the medicaldevice and/or the movable component are to be positioned relative to thepatient and/or relative to the examination room. The nominal positionmarking can for example feature a 3D avatar of the patient, wherein the3D avatar shows the patient in the nominal position of the patient. Thenominal position marking can for example feature a 3D model of themedical device and/or of the movable component, wherein the 3D modelshows the medical device in the nominal position of the medical deviceand/or the movable component in the nominal position of the movablecomponent.

In accordance with one embodiment of the invention region information,which relates to a region of the patient and/or a region of the medicaldevice, is provided, wherein the augmented reality information iscreated based on the region information, wherein the augmented realityinformation features a region marking that marks the region of thepatient and/or the region of the medical device.

The region of the patient can for example feature the structure of thepatient. The region of the patient can for example be a region of thepatient that is to be examined in the medical examination and/or that isselected for the medical examination. The region of the patient can forexample be a region of the patient that is located in the effectiveregion of the medical device and/or that is selected for a transfer intothe action region of the medical device. The action region of themedical device can for example be an action region of an interventiontool, a raw image data acquisition region of a medical imaging device orsimilar. The region of the patient can for example be a region of thepatient in which an intervention is to be carried out and/or from whicha medical imaging dataset is to be recorded. The region of the medicaldevice can for example be a region of the medical device, which is to beused and/or actuated. The region marking can for example feature a 3Dimage of the regions of the medical device and/or a 3D model of theregions of the medical device. This enables the operation of the medicaldevice, for example in the medical examination and/or in a preparationof the medical device, to be planned and/or supported.

In accordance with one form of embodiment of the invention, based on aninput of the first user and/or based on an input of the second user, theregion information can be created and/or the region marking can beadapted in relation to a type, a position, a size and/or a shape. Oneform of embodiment of the invention makes provision for the regioninformation to be established based on an examination plan, based on anexamination protocol of the medical examination, based on the patientposition information and/or based on the nominal position information.

In accordance with one embodiment of the invention a marking, which isselected from the markings group, which consists of the structuremarking, the nominal position marking, the region marking andcombinations thereof, features a representation which is selected fromthe representation group, which consists of a hologram, a hologram-typerepresentation, a stereoscopic representation, a three-dimensionalrepresentation and combinations thereof. This in particular enables a 3Davatar, a 3D image and/or a 3D model to be represented in an especiallyadvantageous manner. A hologram-type representation can especially beunderstood as a visual representation, which gives an observer, forexample the first user and/or the second user, the impression that theyare looking at a hologram.

In accordance with one embodiment of the invention an input of the firstuser is detected, wherein the augmented reality information is createdbased on the input of the first user. In particular a marking, which isselected from the marking group, which consists of the structuremarking, the nominal position marking, the region marking andcombinations thereof, can be created based on the input of the firstuser and/or adapted in relation to a type, a position, a size and/or ashape.

One embodiment of the invention makes provision for the augmentedreality information to be output such that the augmented realityinformation is able to be perceived in a field of view of a second user,that an input of the second user is detected and that the augmentedreality information is created based on the input of the second user.

The augmented reality information is in particular able to be detectedin the field of view of the second user when the augmented realityinformation is visible in field of view of the second user for thesecond user. In particular the augmented reality information can beoutput such that the field of view of the second user is enhanced by theaugmented reality information. In accordance with one form of embodimentof the invention at least a part of the field of view of the first useris shown in a field of view of the second user. In accordance with oneform of embodiment of the invention at least the part of the field ofview of the first user is shown in the field of view of the second usersuch that the second user sees the patient and/or the medical devicefrom the perspective of the first user. In accordance with one form ofembodiment of the invention the augmented reality information is shownin the field of view of the second user such that the second user seesthe augmented reality information from the perspective of the firstuser.

In particular, the features that are described in relation to the firstuser and without reference to the second user also apply correspondinglyin relation to second user, i.e. when in the features the term “firstuser” is replaced by the term “second user” in each case. In particular,a marking that is selected from the marking group which consists of thestructure marking, the nominal position marking, the region marking andcombinations thereof, can be created based on the input of the seconduser and/or adapted in relation to a type, a position, a size and/or ashape. In accordance with one aspect of the invention the nominalposition information and/or the region information can be establishedbased on the input of the first user and/or based on the input of thesecond user.

Preferably the input of the first user is a gesture of the first userand/or the input of the second user is a gesture of the second user.

In accordance with one embodiment of the invention a first user profile,which relates to the first user, and/or a second user profile, whichrelates to the second user, is selected, wherein the augmented realityinformation is created based on the first user profile and/or based onthe second user profile.

The user profile can for example comprise a user category, a function inthe medical examination, an access authorization to patient data of thepatient, a usage authorization for the medical device, an authorizationthat relates to the creation, changing, overwriting and/or removal ofthe augmented reality information, or similar. The user category canrelate for example to a group of people. Optionally a group of peoplecan be assigned respectively to the first user and/or the second user.Groups of people can for example be patients, medical technicians,physicians of different levels of experience, technicians and alsomakers of medical device. The function in an intervention can forexample be selected from a function group consisting of the firstphysician who carries out the intervention on their own, a secondphysician who carries out the intervention under instruction, a thirdphysician who observes the intervention and learns in doing so, and afourth physician who observes the intervention and in doing so givesinstructions for carrying out the intervention.

One form of embodiment of the invention makes provision for theaugmented reality information to be created based on the firstinformation and based on basic information. The basic information canfor example be selected from the basic information group, which consistsof the medical dataset, the examination information, the patientinformation, the medical imaging dataset, the nominal positioninformation, the region information, the input of the first user, theinput of the second user, the first user profile and the second userprofile.

One form of embodiment of the invention makes provision for theaugmented reality information to be created based on the firstinformation and based on a number of items of basic information. Each ofthe number of items of basic information can for example be selected ineach case from the basic information group, which consists of themedical dataset, the examination information, the patient information,the medical imaging dataset, the nominal position information, theregion information, the input of the first user, the input of the seconduser, the first user profile and the second user profile.

In the context of the present application the expression “based on” canbe understood in particular in the sense of the expression “when using”.In particular a formulation in accordance with which a first feature iscreated based on a second feature (alternatively: established,determined etc.) does not exclude that the first feature is created(alternatively: established, determined etc.) based on a third feature.

At least one embodiment of the inventive system for outputting augmentedreality information, which relates to a medical device and/or to amedical examination of a patient, to a first user, features a firstacquisition module, a creation module and a first output module. Thefirst acquisition module is embodied for acquiring first informationthat is selected from the group that consists of image information,depth information, coordinate information and combinations thereof,wherein the first information relates to the medical device and/or themedical examination. The creation module is embodied for creating theaugmented reality information that relates to the medical device and/orthe medical examination of the patient, based on the first information.The first output module is embodied for outputting augmented realityinformation such that the augmented reality information is able to beperceived in a field of view of the first user.

In accordance with one embodiment of the invention the system featuresthe medical device. Preferably the medical device is embodied forproviding the coordinate information.

In accordance with one embodiment of the invention the medical devicefeatures a movable component.

In accordance with one form of embodiment of the invention the systemand/or the medical device features a control device that is embodied foroutput of a control command, which relates to a movement of the movablecomponent. In accordance with one form of embodiment of the inventionthe system features a movement parameter acquisition unit that isembodied for acquiring a movement parameter that relates to a movementof the movable component. The movement parameter acquisition unit canfor example feature a sensor. The movement parameter can be acquired forexample via the movement parameter acquisition unit.

One form of embodiment of the invention makes provision for the movablecomponent to be able to be controlled, in particular positioned and/ororiented, via the first augmented reality device and/or via the secondaugmented reality device.

In accordance with one form of embodiment of the invention the medicaldevice features a drive unit that is embodied to drive a movement of themovable component. Preferably the drive unit is embodied for receivingthe control command and/or for driving a movement of the movablecomponent based on the control command. The drive unit can be embodiedas an electric motor for example. The movement parameter can for examplespecify an electric charge consumed by the drive unit when driving themovement and/or electric energy consumed by the drive unit when drivingthe movement.

The movement parameter acquisition unit can be integrated for exampleinto the control device and/or into the drive unit. The movementparameter acquisition unit can be embodied for example for measuring themovement parameter. The movement parameter can for example specify arotational speed angle, in particular a number of revolutions. Themovement parameter can for example be a measure for the movement of themovable component and/or relate to the movable component, the drive unitand/or the movement parameter acquisition unit. In particular acalibration and/or a simulation of the medical device and/or of thecontrol device and/or of the movable component and/or the drive unitand/or a movement of the movable component can be carried out.

Preferably the system features an establishment unit, which is embodiedfor establishing the coordinate information based on the control commandand/or based on the movement parameter. The coordinate information canfor example be established via the establishment unit based on thecontrol command and/or based on the movement parameter.

In accordance with one form of embodiment of the invention the firstacquisition module features a first camera unit, wherein the firstcamera unit is embodied for acquiring the image information and/or thedepth information. In accordance with a further form of embodiment ofthe invention the first acquisition module can access a memory area ofan information provision module, wherein the first information isacquired via a data transfer from the information provision module tothe first acquisition module. In accordance with one form of embodimentof the invention the system features the information provision module.The information provision module is embodied for providing the firstinformation.

In accordance with one form of embodiment of the invention the firstoutput module is embodied for outputting the augmented realityinformation such that the field of view of the first user is enhanced bythe augmented reality information. In accordance with one form ofembodiment of the invention the first output module features a firstviewing device, in particular in the form of data eyeglasses, a datacontact lens and/or a retinal projector. In particular the augmentedreality information can be output by way of the first viewing device. Inaccordance with a further form of embodiment of the invention a firstviewing device can access a first memory area of the first outputmodule, wherein the augmented reality information will be output via adata transfer by the first output module to the first viewing device. Inaccordance with one form of embodiment of the invention the systemfeatures the first viewing device.

In accordance with one embodiment of the invention the system featuresthe medical device, wherein the medical device is embodied for provisionof a medical dataset. Preferably the creation module is embodied forcreating the augmented reality information based on the medical dataset.

In accordance with one embodiment of the invention the system features amedical imaging device, wherein the medical imaging device is embodiedfor provision of a medical imaging dataset, which relates to a structureof the patient. Preferably the creation module is embodied for creatingthe augmented reality information based on the medical imaging dataset.Preferably the augmented reality information features a structuremarking, which marks the structure of the patient.

In accordance with one embodiment of the invention the system features afirst input module, which is embodied for acquiring an input of thefirst user, and/or a second input module, which is embodied foracquiring an input of the second user. Preferably the creation module isembodied for creating the augmented reality information based on theinput of the first user and/or based on the input of the second user.

One form of embodiment of the invention makes provision for the firstacquisition module to feature the first input module and/or for thefirst acquisition module to be embodied for acquiring an input of thefirst user and/or for the first camera unit to be embodied for acquiringthe input of the first user. In accordance with one aspect of theinvention the system features a second output module and a secondacquisition module. The second output module is embodied for output ofthe augmented reality information such that the augmented realityinformation is able to be perceived in a field of view of a second user.One form of embodiment of the invention makes provision for the secondacquisition module to feature the second input module and/or for thesecond acquisition module to be embodied for acquiring the input of thesecond user and/or for the second camera unit to be embodied foracquiring the input of the second user.

In accordance with one form of embodiment of the invention the secondoutput module is embodied for showing at least a part of the field ofview of the first user in a field of view of the second user. Inaccordance with one form of embodiment of the invention the secondoutput module is embodied for output of the augmented realityinformation such that the field of view of the second user is enhancedby the augmented reality information.

In accordance with one form of embodiment of the invention the secondoutput module features a second viewing device, in particular in theform of data eyeglasses, a data contact lens and/or a retinal projector.In particular the augmented reality information can be output by way ofthe second viewing device. In accordance with a further form ofembodiment of the invention a second viewing device can access a memoryarea of the second output module, wherein the augmented realityinformation will be output via a data transfer by the second outputmodule to the second viewing device. In accordance with one form ofembodiment of the invention the system features the second viewingdevice.

In accordance with one form of embodiment of the invention the secondacquisition module features a second camera unit, wherein the secondcamera unit is embodied for acquiring the input of the second user. Inaccordance with a further form of embodiment of the invention the secondacquisition module can access a memory area of an input provisionmodule, wherein the input of the second user is acquired via a datatransfer from the input provision module to the second acquisitionmodule. In accordance with one form of embodiment of the invention thesystem features the input provision module.

In accordance with one embodiment of the invention the system features afirst augmented reality device, which features the first acquisitionmodule and/or the first output module. The first augmented realitydevice can for example be embodied as data eyeglasses and/or a dataheadset. The first augmented reality device can for example be wearableand/or worn by the first user. In accordance with one form of embodimentof the invention the first augmented reality device features thecreation module.

In accordance with one embodiment of the invention the system features asecond augmented reality device, which features the second acquisitionmodule and/or the second output module. The second augmented realitydevice can for example be embodied as data eyeglasses and/or a dataheadset. The second augmented reality device can for example be wearableand/or worn by the second user. In accordance with one form ofembodiment of the invention the second augmented reality device featuresthe creation module.

One form of embodiment of the invention makes provision for the creationmodule to feature a first creation submodule and a second creationsubmodule, for the first augmented reality device to feature the firstcreation submodule and for the second augmented reality device tofeature the second creation submodule.

One form of embodiment of the invention makes provision for theaugmented reality information to be created via a machine-learningalgorithm. The machine-learning algorithm can for example evaluate thefirst information, the examination information, the patient information,the nominal position information, the region information, the input ofthe first user and/or the input of the second user.

In accordance with one embodiment of the invention the system features auser profile selection module. The user profile selection module isembodied for selection of a first user profile, which relates to thefirst user, and/or for selection of a second user profile, which relatesto the second user. Preferably the creation module is embodied forcreation of the augmented reality information based on the first userprofile and/or based on the second user profile. The user profileselection module can for example provide a graphical user interface,wherein the first user profile and/or the second user profile can be setup and/or selected by way of the graphical user interface. In accordancewith one form of embodiment of the invention the user profile selectionmodule has a security interface for acquiring a security certificate.The security certificate can in particular be a password and/or ahardware key, for example a smart card. In particular this enablesaccess to the first user profile and/or access to the second userprofile to be protected.

In accordance with one embodiment of the invention the system isembodied for carrying out an inventive method for outputting augmentedreality information, which relates to a medical device and/or a medicalexamination of a patient.

In particular, an embodiment of the invention make communication betweena number of users possible, in particular improving communicationbetween a number of users from different groups of people that areinvolved in using a medical device and/or involved in a medicalexamination of a patient. For a user of a device for creating anaugmented reality, additional (audio)-visual elements can be added tothe real world that he or she perceives in their field of vision,existing elements can be edited out and/or changed in their appearance.In addition images and sound can be picked up from the perspective ofthe user and/or the direction of view of the user can be determined.

This information can be provided to one or more further users of anotherdevice in each case for creating an augmented reality, in particular asaugmented reality information. In this case the augmented realityinformation can be provided at the same location or provided at anotherremote location by remote data transmission. The augmented realityinformation and/or the created augmented reality can be used in order tosupport communication between the users involved in using a medicaldevice, for example a medical imaging device, and/or involved in amedical examination. The user can for example be selected from the usergroup that consists of the first user and the second user. The user orthe number of further users can be selected in each case from the usergroup that consists of the first user and the second user.

One application of medical imaging lies in the field of intervention, inwhich the images provided by the medical imaging device are used duringa medical intervention in order to plan the intervention and to checkthe progress of the intervention. This type of use of the medicalimaging device in many cases makes high demands on the physician andrequires a high level of training, in which typically an experiencedphysician passes on his or her knowledge to a more inexperiencedphysician. The communication between the two physicians is able in thiscase to be supported by a device for creating an augmented reality sothat the experienced physician observes the images recorded from theperspective of the inexperienced physician and can insert tips as to howto proceed further directly into the field of view of the inexperiencedphysician. In this case the inexperienced physician wears a device forcreating an augmented reality, while the experienced physician eitherlikewise wears such a device or uses a conventional device forinformation processing, e.g. a laptop or a tablet. One form ofembodiment of the invention makes provision for the first input moduleand/or the second input module in each case to feature a mouse, ajoystick, a touchpad, a touch-sensitive input device and/or similar.

The experienced physician could for example, during an intervention forablation, display the nominal position of the ablation needle on thepatient and thus instruct the inexperienced physician. Likewise theroles could be reversed and the experienced physician could undertakethe intervention. Then the inexperienced physician could observe theintervention from the perspective of the experienced physician and thuslearn by example. In this case it is of no significance whether theexperienced and inexperienced physician are located in the same room, sothat this instruction can also be done “remotely”, i.e. by remotetransmission. To this end it is useful for there also to be abidirectional transmission of audio signals as well as the transmissionof the augmented reality information, so that the two physicians cantalk to each other. It is not absolutely necessary for the inexperiencedphysician to place himself or herself in the perspective of theexperienced physician simultaneously with the carrying out of theintervention. For example the first information and/or the augmentedreality information can be recorded and/or enhanced with additionalinformation inserted during the medical examination or retrospectivelyand made available to the inexperienced physician at a later time.

One embodiment of the invention makes provision for the support of aninexperienced physician by an experienced physician to take place not inthe sense of training but during normal operation. Thus an experiencedphysician could be available on demand as an expert in order to supportphysicians, e.g. in rural regions, when complications occur or inespecially complicated cases. In particular the experts could support aphysician in carrying out biopsies supported by imaging. This supportcould also be provided in advance of the intervention, in order todetermine the best possible access for the intervention. For example theexpert could assist the physician in the planning of the interventionpath on the basis of the medical imaging dataset.

An embodiment of the invention in particular makes it possible tosupport the conversation between the patient and a physician. This cantake place before an intervention for example, wherein the physicianexplains to the patient in the augmented reality and/or by way of theaugmented reality information the execution or the results of theintervention or visualizes the risks. Likewise the physician can showthe patient the results of the examination in the augmented realityand/or by way of the augmented reality information after theintervention has been performed.

In such cases it can be advantageous to show these to the patient in a“virtual mirror”. It is actually also conceivable to show the patientthe result of an examination by way of inserting the augmented realityinformation directly at the examined point on their body. However on theone hand this can be complicated for the patient or not able to be seenat all (e.g. for the spinal column). On the other hand it can be verydisconcerting for the patient if this representation of the result istoo realistic. Therefore it can be advantageous not to insert thisrepresentation of the result on the patient himself or herself, but on a“virtual mirror”, i.e. a surface within their augmented reality, whichshows the patient similar to a mirror image and into which the result ofthe examination can be inserted. As an alternative or in addition amonitor together with a camera can be used for this representation ofthe result.

An embodiment of the invention makes improved instruction of the patientbefore and/or during the medical examination possible for example. Forthe patient the medical examination, which can be carried out with themedical device for example, typically represents an exceptionalsituation. In general the patient possesses little knowledge about theprocedures in the clinic and during the examination.

Communication between the clinic personnel and the patient can besupported by the use of a device for creating an augmented reality. Tothis end the patient can for example be guided through the clinic to thelocation of the examination by augmented reality information insertedinto his or her field of view, e.g. arrows on the floor. At the locationof the examination he or she can be shown by way of the augmentedreality information, in particular in the form of audiovisualinformation, whether he or she still has to wait, how many patientsthere are before him or her, how long the waiting time is likely to beand which examination room he or she should go to for the examination.Within the framework of the preparation for the examination he or shecan be instructed by way of the inserted augmented reality information,in particular in the form of audiovisual information, to take off his orher clothing in accordance with the examination and/or to put on otherclothing. The patient can further be instructed to take up a particularpose in the examination room, e.g. to lay down in a particularorientation on the examination couch and/or to raise his or her arms.This can also be supported by a “virtual mirror” in which the patientsees himself or herself and/or an avatar of himself or herself in amirror generated in the augmented reality. After a part step of theexamination he or she can be instructed as to the way in which he or shecan leave the examination room and/or be guided for a further part stepof the examination, for example in another examination room.

A particular advantage of the use of a device for creating an augmentedreality lies in the fact that the augmented reality information can betailored to the individual patients and a number of patients in the sameroom can each perceive their individual augmented reality information,e.g. for the instruction as to which examination room they should go to.A further advantage lies in the fact that confidentiality is betterpreserved when each patient can only perceive the instructions that areintended for him or her and e.g. calling the patient by name in thewaiting room can be dispensed with.

An embodiment of the invention further makes possible improved supportof personal communication between the patient and the operatingpersonnel of a medical device. During a CT or MRT examination it isoften not possible for the operating personnel to remain with thepatient all the time. Instead, with CT for reasons of radiationprotection for example, they withdraw from the examination room into acontrol room. By way of augmented reality information the presence ofthe operating personnel in the examination room, for examplesimultaneously with a bidirectional sound transmission, can be shown forthe patient. By way of augmented reality information the patient can beshown as an element of the augmented reality within the control room forthe operating personnel.

The bidirectional sound transmission can be undertaken by way of anintercom system for example. As an alternative or in addition thebidirectional sound transmission can be modified in real time, e.g. byin particular simultaneous machine translation. In this way the patientand the operating personnel can understand one another even if they donot speak a common language.

Through these applications of augmented reality the patient does notfeel left alone during the examination and his or her anxieties can bereduced. It is possible in particular, by way of the augmented realityinformation, to display the operating personnel embodied by an avatarfor the patient in the examination room. The avatar could e.g. bepresent constantly. The operating personnel can for example only belinked to the avatar when a patient situation demanding attention, forexample noises and/or movements, is detected via sensors. In this waythe operating personnel can go about their work, while for the patienton the other hand the feeling of a constant presence is produced, whichhelps to reduce his or her tension and anxieties.

An embodiment of the invention in particular makes it possible tosupport the operating personnel, e.g. in the placement of the patient,in the selection of support aids or in the placement of EKG electrodesor devices for measuring the breathing. In such cases this support canbe provided on the one hand by another person, e.g. an applicationspecialist of the maker, who inserts information for placement and/orselection for the operating personnel by way of the augmented realityinformation. On the other hand the augmented reality information can becreated automatically for example on the basis of the specifications ofthe maker, which can be adapted additionally to the specificrequirements of the individual clinic, e.g. on the basis of the selectedmeasurement protocol of the medical device or on the basis of otherdata. This other data can be anatomical landmarks of the patient forexample. Anatomical landmarks can be established in particular based onthe image information and/or based on the depth information. The optimumposition of EKG electrodes can be computed via anatomical landmarks forexample.

Thus the communication from the maker to the user, which would otherwisebe undertaken above all by training and by the operating instructions,can be supported by way of the augmented reality information.

An embodiment of the invention in particular makes possible improvedtraining of the operating personnel. To this end for example a virtualpatient can be introduced by way of augmented reality information intothe augmented reality of the operating personnel to be trained, whereinan examination can be carried out as a exercise with the virtualpatient. The virtual patient in this case can be fullycomputer-generated, i.e. also generated by inputs of a further person,e.g. likewise with the aid of a device for creating an augmentedreality. This procedure offers the advantage, in particular with x-raydevices, that in this way, even when conducting a training exercise, thetriggering of the x-ray radiation can be simulated. Conducting trainingexercises with real patients, in particular because of the radiationload, would be associated with severe restrictions.

In accordance with one form of embodiment of the invention the firstaugmented reality device features a first camera unit, wherein the firstcamera unit is embodied for acquiring the image information and/or thedepth information.

One form of embodiment of the invention makes provision for the firstaugmented reality device to be embodied as data eyeglasses. Such dataeyeglasses can be worn by the user for example. The user can for examplebe a radiologist or an assistant medical technician. In one form ofembodiment of the invention an image of the patient is acquired via thefirst camera unit. This image is compared with entries in a patientdatabase. This enables it to be established which patient is involvedand/or whether there is a match with the planned examination. Such anidentification check enables incorrect examinations and incorrectassignments of examination results to be avoided.

In a further variant of an embodiment of the invention information,annotations and/or images from a pre-examination of the patient aretransferred to the user by way of the augmented reality informationand/or made available in an augmented reality.

By a clever presentation and an interactive interaction with the datapresented by way of the augmented reality information (images,histology, blood values etc.) the user can very quickly generate aholistic overall picture of the patient.

Based on the patient information, in particular based on the patientidentification information, patient history information that relates toa patient history of the patient can be established.

Based on the patient information for example characteristic features ofthe patient can be established. For example with the aid of a databasein which reference datasets are stored, based on the patientinformation, a suitable reference dataset can be established, which canbe used for example for dose modulation in a medical examination withionizing radiation.

A medical image, which was established for example within the frameworkof a pre-examination, for example as a 3D model of the body region ofthe patient, which corresponds to the medical image, can be shownprojected-on and/or overlaid by way of the augmented realityinformation. In this way the user can recognize quickly and accuratelywhether diagnostically relevant information is already present at aparticular, for example painful, point.

One form of embodiment of the invention makes provision for the patientto be recognized based on the image information, for patient informationand/or patient data to be established based on the first information andto be made available by way of the augmented reality information. Thatcan be advantageous in particular when an emergency physician isdeployed to the scene of an accident.

In accordance with a further form of embodiment of the invention theaugmented reality information is presented location-specifically in thefield of view of the user. For example a diagnostic display, which isarranged in an examination room, in particular an OP room, can beenhanced by the augmented reality information. The augmented realityinformation can for example feature an EKG signal, a breath curve, apatient history, biological data of the patient. The augmented realityinformation can in particular be output during viewing of the medicaldevice and/or of the diagnostic display.

In another form of embodiment of the invention the user, for example aradiologist, an assistant medical technician or a nursing sister, willbe equipped with a device for creating an augmented reality, e.g. in theform of data eyeglasses. Preferably a unique feature, able to beevaluated automatically, for example a QR code, is arranged on thepatient and/or on the patient file. When this feature is looked at avirtual ticket can be shown as augmented reality information forexample. The user can thus see the current information that relates tothe patient.

A further form of embodiment of the invention makes provision for theuser to be able to log on to an account generated for the user. With theaccount a user profile of the user, which in particular includes rightsof the user, can be managed for example. In this way exclusively thatinformation and/or those images for which the user is authorized and/orwhich relate to a patient assigned to the user can be output, inparticular by way of augmented reality information.

The first information can for example feature a topogram, in particulara camera-based topogram. In particular the topogram can be a combinationof image information and depth information. In accordance with one formof embodiment of the invention a camera-based topogram is acquired via afirst camera unit, which is integrated for example into the firstaugmented reality device. The camera-based topogram can for example beshown overlaid by way of the augmented reality information on thepatient. Above and beyond this characteristic features of the patientcan be established based on the camera-based topogram. With the aid of adatabase in which reference datasets are held, a suitable reference datafor the medical examination can thus be established based on thecamera-based topogram, which can be used for example for dosemodulation.

In accordance with one form of embodiment of the invention patientstatus information is transferred to the user by way of the augmentedreality information. The patient status information can for example be afacial expression of the patient, a body temperature of the patient, apupil dilation of the patient and/or similar. The patient statusinformation can for example relate to emotional indicators, which inparticular can point to stress of the patient. By way of augmentedreality information for example one or more physiological parameters ofthe patient can be displayed, in particular a pulse, a breathing rhythmand/or similar.

One form of embodiment of the invention makes provision for the patientinformation, for example immediately before an intervention, to beacquired via the data eyeglasses, for the acquired patient informationto be compared with entries in a database, for a matching entry from thedatabase, for example based on similar bodily features of the patientand/or based on a similar intervention, to be established and for anintervention result assigned to the matching entry to be shown projectedand/or overlaid onto the surface of the patient. In this way a betterimpression of the result of the intervention to be expected can beimparted to the user, in particular a radiologist and/or surgeon.

In accordance with a further form of embodiment of the invention thepatient is equipped with data eyeglasses. Physiological parameters ofthe patient, such as e.g. heartbeat, which allow an impression of theemotional state of the patient to be obtained, can be acquired. If thesephysiological parameters point to the patient being under stress, acalming environment serving to relieve stress can be displayed via thedata eyeglasses. For example countryside that suggest wide-open spacescan be shown to a patient who suffers from claustrophobia to be examinedwith a medical imaging device. As an alternative or in additionexamination-relevant instructions, e.g. breathing commands, can bedisplayed via the data eyeglasses. Furthermore audio signals can also beused.

In accordance with one form of embodiment of the invention virtual, forexample computer-generated, data, which relates to the medicalexamination, can be shown overlaid on the patient by way of theaugmented reality information. Thus a better ability to conceive andcarry out a medical examination is especially able to be realized.

With a device for creating an augmented reality, in particular medicalimages, e.g. DICOM images, and additional information can be shownoverlaid on the patient by way of the augmented reality information. Theuse of a device for creating an augmented reality for a medicalexamination, for example for an intervention, makes possible enormousimprovements in particular in relation to the visualization, the displayof information and the provision of virtual processing tools, inparticular image processing tools.

The medical imaging dataset can for example feature a medical image, inparticular a DICOM image, a CT image, an MRT image, an ultrasound image,an angiography image series, an endoscopy video or similar. Theaugmented reality information can feature the medical imaging datasetand/or a marking that was established based on the medical imagingdataset. Thus the medical imaging dataset and/or the marking can beshown overlaid on the patient in the field of view of the user. In thiscase it is possible to align a structure marking shown and/orestablished via the medical imaging dataset with the correspondingstructure of the patient.

In this way the user who is carrying out the medical examination canaccess information from the medical imaging dataset more quickly,ergonomically more conveniently and with a reduced susceptibility toerrors.

The device for creating an augmented reality can be embodied to create avirtual screen in the field of view of the user. In particular thevirtual screen can be available in any position and direction of view ofthe user. That is especially advantageous in relation to ergonomicaspects. In addition the restriction of the user's freedom of movementby monitors and/or the support structures is reduced.

The augmented reality information can for example feature the virtualscreen. The virtual screen can for example be displayed or edited out asrequired. A medical dataset, in particular a medical image, examinationinformation and/or patient information can be displayed in the virtualscreen for example.

One form of embodiment of the invention makes provision for a structureof the patient to be selected based on the medical imaging dataset andfor the augmented reality information to feature a structure marking,which marks the selected structure of the patient. In this way theidentification of small and difficult-to-find structures of the patientis made easier.

The structure marking can for example mark an intervention area. Thestructure marking can further feature a representation of anintervention planning model, which relates to a planned intermediatestep or to a planned result of the intervention. The interventionplanning model can be established for example via a simulation. With theaid of the augmented reality information the intervention planning modelcan be shown overlaid on the patient. Navigation during the interventioncan be improved with the intervention planning model. For example theintervention planning model can feature a planned instrument path. Inthis way it is also made possible for a user who has less experience inrelation to a given intervention to carry out the given interventionsuccessfully.

The intervention planning model can feature a model of an implant. Theintervention planning model can be adapted for example via gestureinformation of the user to the structure of the patient and/or to ananatomical feature of the patient. In this way an optimum position, sizeand shape of the implant are able to be established.

The structure marking can for example feature a representation of asegmentation of a structure, in particular of an organ, of the patient.By way of augmented reality information for example diagnosticinformation, which relates to a result of a diagnosis can be shownoverlaid on the patient. It is thus made possible for the user torecognize critical regions of the structure and to avoid damage duringthe intervention.

In accordance with one form of embodiment of the invention the patientstatus information is output by way of the augmented realityinformation. The patient status information can for example relate to aheartbeat of the patient, breathing of the patient, a patient history ofthe patient, a laboratory value of the patient, a demographicspecification of the patient or similar. In this way the patient statusinformation can be output in the field of view of the user. Thus animproved basis for a decision can be provided for the user, inparticular a physician who is examining the patient.

The augmented reality information can feature a virtual processing tool,in particular a virtual image processing tool. For example a region ofthe patient can be marked by way of the virtual processing tool and/ormeasured by way of the virtual processing tool. In the measurement ofthe region of the patient for example, a position, an orientation, aform, a length, a surface, a volume of the region or similar can beestablished.

The augmented reality information can for example feature diagnosticinformation. The diagnostic information can be provided for example viaa PACS dataset and/or via a CAD dataset. The diagnostic information canfor example be established based on the medical imaging dataset. Themedical imaging dataset can for example feature a rendered image(rendering image) and/or an unfolded image (unfolding image) of thestructure of the patient, for example of a skull, of a rib cage, inparticular one or more ribs, etc. The rendered image can in particularbe based on user-specific rendering.

The nominal position information can for example be established based ona position of the patient during a pre-examination and/or treatment, inparticular a radiation therapy treatment, which has taken place at atime before the medical examination. The reproducibility of the positionof the patient can thus be improved.

In accordance with one form of embodiment of the invention the medicaldevice is selected from the group that consists of a medical imagingdevice, an intervention tool, an examination device that is embodied foracquisition and/or for provision of a medical dataset, and combinationsthereof.

The features that are described in relation to the medical device alsoapply accordingly in relation to a virtual medical device, i.e. when theterm “medical device” is replaced by the term “virtual medical device”in each case in the features. The features that are described inrelation to the medical examination apply accordingly in relation to avirtual medical examination, i.e. when the term “medical examination” isreplaced by the term “virtual medical examination” in each case in thefeatures. The features that are described in relation to the patientapply accordingly in relation to a virtual patient, i.e. when the term“patient” is replaced by the term “virtual patient” in each case. Inthis way training of medical personnel and/or planning of the medicalexamination can be improved. In particular a realistic training and/or arealistic planning is thus made possible even without the use of realpatients and/or real medical devices.

One form of embodiment of the invention makes provision for the firstuser to be able to perceive their environment by way of the firstaugmented reality device. In particular an image of the patient and/oran avatar of the patient can be displayed to the first user instead ofthe real patient and/or an image of the medical device and/or a model ofthe medical device can be displayed to the user at the correct locationinstead of the real medical device. In this way a device for creating avirtual reality can also be used as an augmented reality device. A knowndevice for creating a virtual reality is e.g. Rift from Oculus VR, LLC,USA.

One form of embodiment of the invention makes provision for theinventive system and/or for a number of components of the inventivesystem to be realized at least partly in the form of software on aprocessor system. In particular the first acquisition module, thecreation module, the first output module, the information provisionmodule, the second output module and the second acquisition module caneach form a component of the inventive system and/or each be realized atleast partly in the form of software on a processor system. One form ofembodiment of the invention makes provision for the inventive systemand/or a or a number of components of the inventive system to berealized at least partly in the form of software-supported hardware, forexample FPGAs, a processor system or the like.

One form of embodiment of the invention makes provision for theinventive system and/or for a number of components of the inventivesystem to be formed at least partly by a Cloud via Cloud Computing. Oneform of embodiment of the invention makes provision for the creationmodule and/or at least a submodule of the creation modules to be formedby a Cloud via Cloud Computing. The Cloud can in particular feature anetwork of memory areas spatially remote from one another and processorsystems spatially remote from one another. For data transfer from theCloud and/or to the Cloud the first augmented reality device can featurea first Cloud interface and/or the second augmented reality device canfeature a second Cloud interface.

Data can be transferred between components of the system e.g. via asuitable interface. One form of embodiment of the invention makesprovision for interfaces for data transfer to and/or from components ofthe inventive system to be realized at least partly in the form ofsoftware. In particular the interfaces can have access facilities tosuitable memory areas in which data can be suitably buffered, retrievedand updated. The interfaces can also be embodied as interfacesconstructed from hardware, which can be activated by suitable software.

A largely software-based realization of an embodiment of the inventivesystem has the advantage that even augmented reality devices previouslyalready used can be upgraded by a software update, in order to work inthe inventive way. To this extent the object is also achieved by acorresponding computer program product with a computer program, which isable to be loaded into the memory device of a computer, wherein thesteps of the inventive method can be carried out with the computerprogram when the computer program is executed on the computer. Such acomputer program product, as well as the computer program, can includeadditional software components, e.g. documentation, and/or hardwarecomponents, e.g. a hardware key (dongle etc.) for use of the software.

For transport of the computer program and/or for storage of the computerprogram to or in an augmented reality device a computer-readable medium,for example a memory stick, a hard disk or another transportable orpermanently installed data medium can be use, on which a computerprogram is stored, which is able to be loaded into a memory device of acomputer, wherein the steps of an inventive method will be carried outwith the computer program, when the computer program is executed on thecomputer. One form of embodiment of the invention makes provision forthe first augmented reality device to feature a first computer and/orfor the second augmented reality device to feature a second computer.

The first computer and/or the second computer can each feature aprocessor system, which e.g. features a microprocessor or a number ofmicroprocessors working together.

In accordance with one embodiment of the invention the medical imagingdevice is selected from the group that consists of a C-arm x-ray device,a computed tomography device (CT device), a Single Photon EmissionComputed Tomography device (SPECT device), a Positron EmissionTomography device (PET device), a Magnetic Resonance Tomography device(MRT device) and combinations thereof. In particular the medical imagingdevice can feature an x-ray device, an ultrasound device and similar.The medical imaging device can further be a combination of a number ofimaging and/or irradiation modalities. In such cases an irradiationmodality can feature an irradiation device for therapeutic irradiationfor example.

Within the framework of an embodiment of the invention features that aredescribed in relation to different forms of embodiment and/or differentclaim categories (method, system etc.), can be combined into furtherforms of embodiment. In particular the features, advantages and forms ofembodiment described in relation to the inventive method are also to betransferred to the inventive system, the inventive computer programproduct and the inventive computer-readable medium and vice versa. Inother words the device claims can also be further developed with thefeatures that are described or claimed in conjunction with a method.Functional features of an inventive method can in such cases also becarried out by correspondingly embodied components or modules of theinventive system. The use of the indefinite article “a” or “an” does notexclude features involved also being present a number of times. It ispossible for a module to feature a number of submodules separatedphysically from one another and for a unit to be able to feature anumber of subunits separated physically from one another.

FIG. 1 shows a flow diagram of a method for outputting augmented realityinformation AR, which relates to a medical device 2, IT, MD and/or amedical examination of a patient 13, to a first user U1 in accordancewith a first form of embodiment of the invention. In the step R1 firstinformation is acquired that is selected from the group that consists ofimage information, depth information, coordinate information andcombinations thereof, wherein the first information relates to themedical device 2, IT, MD and/or the medical examination of the patient13. In the step GA augmented reality information AR, which relates tothe medical device 2, IT, MD and/or the medical examination of thepatient 13, is created based on the first information. In the step D1the augmented reality information AR is output such that the augmentedreality information AR is able to be perceived in the field of view ofthe first user U1.

FIG. 2 shows a diagram of a system 1 in accordance with a second form ofembodiment of the invention. The system 1 is embodied for outputtingaugmented reality information AR, which relates to a medical device 2,IT, MD and/or a medical examination of a patient 13, to a first user U1and features a first acquisition module R1-M, a creation module GA-M anda first output module D1-M. The system 1 is in particular embodied forcarrying out a method in accordance with the first form of embodiment ofthe invention. In particular the detection R1 of the image informationcan be carried out via the first acquisition module R1-M, the creationGA of the augmented reality information AR via the creation module GA-Mand the output D1 of the augmented reality information AR via the firstoutput module D1-M.

FIG. 3 shows a system 1 in accordance with a third form of embodiment ofthe invention. The system 1 features a first augmented reality deviceH1. The first augmented reality device H1 is embodied as dataeyeglasses. The first augmented reality device H1 features a firstcamera unit C1, a first eye tracking unit E1, a first viewing device V1and a first control unit N1. The first control unit N1 is a firstcomputer, in particular a first digital computer, and is embodied forcontrolling the first augmented reality device H1. The first controlunit N1 features the first acquisition module R1-M, the creation moduleGA-M and the first output module D1-M. The first control unit N1features a first memory unit M1 and a processor system. The first memoryunit M1 is embodied for loading a computer program, wherein the steps ofan inventive method are carried out with the computer program when thecomputer program is executed on the first control unit N1. The firstacquisition module R1-M, the creation module GA-M and the first outputmodule D1-M are each realized in the form of software on the processorsystem of the first control unit N1. The image information and/or thedepth information can be recorded via first camera unit C1. The imageinformation and/or the depth information can be acquired via a datatransfer from the first camera unit C1 to the first acquisition moduleR1-M. The first viewing device V1 can access a memory area of the firstoutput module D1-M, wherein the augmented reality information can beoutput via a data transfer from the first output module D1-M to thefirst viewing device V1. The first eye tracking unit E1 is embodied foracquiring eye tracking of the first user U1. This especially enables aposition and/or an orientation of the field of view of the first user U1to be determined relative to a map dataset. In this way a positionand/or an orientation relative to the field of view of the first user U1can be determined for a real object, for example the patient 13, and/ora virtual object, for example the structure marking MI, based on aposition and/or an orientation relative to the map dataset.

FIG. 4 shows a flow diagram of a method for outputting augmented realityinformation AR, which relates to a medical device 2, IT, MD and/or amedical examination of a patient 13, to a first user U1 in accordancewith a fourth form of embodiment of the invention. In step D2 theaugmented reality information AR is output such that the augmentedreality information AR is able to be perceived in a field of view of asecond user U2. In step R2 an input of the second user U2 is acquired.In this case the augmented reality information AR is created based onthe input of the second user U2.

FIG. 5 shows a diagram of a system 1 in accordance with a fifth form ofembodiment of the invention. The system 1 features the first acquisitionmodule R1-M, the creation module GA-M, the first output module D1-M, asecond output module D2-M and a second acquisition module R2-M. Thesystem 1 is in particular embodied for carrying out a method inaccordance with the fourth form of embodiment of the invention. Inparticular the outputting D2 of the augmented reality information AR canbe carried out via the second output module D2-M and the acquisition R2of the input of the second user U2 via the second acquisition moduleR2-M.

FIG. 6 shows a system 1 in accordance with a sixth form of embodiment ofthe invention. The system 1 features a medical imaging device 2 with anacquisition area 4 formed by a tunnel-shaped opening. Withoutrestricting the general inventive thinking, a computed tomography device1 is shown by way of example for the medical imaging device 2.

The medical imaging device 2 features a gantry 20, the acquisition area4, the patient support device 10, a raw image data acquisition device26, 28 and a control device 30. The gantry 20 features a stationarycarrier frame 21 and a rotor 24. The rotor 24 is supported to allowrotation about an axis of rotation via a rotational support device. Theacquisition area 4 is formed by a tunnel-shaped opening in the gantry20. A region of an object, in particular of the patient 13, to be imagedis able to be arranged in the acquisition area 4. The patient supportdevice 10 features a support table 11 for supporting the patient 13,wherein the transfer plate 12 is arranged movably relative to thesupport table 11 on the support table 11 such that the transfer plate 12is able to be introduced into the acquisition area 4 in a longitudinaldirection of the transfer plate 12.

The raw image data acquisition device 26, 28 is a projection dataacquisition device 26, 28 with a radiation source 26, e.g. an x-raysource, and a detector 28, e.g. an x-ray detector. The radiation source26 is arranged on the rotor 24 and is embodied for emission ofradiation, e.g. x-ray radiation, with x-ray quanta 27. The detector 28is arranged on the rotor 24 and is embodied for detection of the x-rayquanta 27. The x-ray quanta 27 can reach the area of the patient 13 tobe imaged from the radiation source 26 and can strike the detector 28after interacting with the area to be imaged. In this way projectiondata of the area to be imaged can be acquired. The projection dataacquired by the projection data acquisition device 26, 28 is passed onto the control device 30. The control device 30 is embodied forcontrolling the medical imaging device 2. The control device 30 featuresan image reconstruction device 34. A medical imaging dataset can bereconstructed means of the image reconstruction device 34 based on theprojection data. The medical imaging device 2 is embodied for provisionof a medical imaging dataset that relates to a structure SP of thepatient 13.

The medical imaging device 2 is embodied for provision of the coordinateinformation. The medical imaging device features a movable component 12.The control device 30 is embodied for output of a control command, whichrelates to a movement of the movable component 12. Without restrictingthe general inventive thinking, the transfer plate 12 is shown for themovable component 12. The support table 11 features a drive unit anelectric motor, which is embodied for driving a movement of the transferplate 12. The drive unit is embodied for receiving the control commandand/or for driving a movement of the transfer plate 12 based on thecontrol command. The control device 30 features a coordinate informationprovision unit 50 an establishment unit 51 and a movement parameteracquisition unit 52. The coordinate information provision unit 50 isembodied for providing the coordinate information. The movementparameter acquisition unit 52 is embodied for acquiring of a movementparameter, which relates to a movement of the movable component 12. Theestablishment unit 51 is embodied for establishing the coordinateinformation based on the control command and/or based on the movementparameter.

The medical imaging device 2 features an input device 38 and an outputdevice 39. The input device 38 is embodied for inputting the controlinformation, e.g. image reconstruction parameters and/or examinationparameters. The output device 39 is for outputting control informationand/or images.

The system 1 features the first augmented reality device H1 with thefirst control unit N1 and a second augmented reality device H2 with asecond control unit. The second augmented reality device H2 is embodiedas data eyeglasses. The structure and functioning of the secondaugmented reality device H2 essentially correspond to the structure andfunctioning of the first augmented reality device. Instead of the firstacquisition module R1-M of the first augmented reality device H1 thesecond augmented reality device H2 features the second acquisitionmodule R2-M. The second control unit, instead of the correspondingcomponents of the first augmented reality device H1, features the secondoutput module D2-M, a second camera unit, a second memory unit and asecond viewing device V2. The sixth form of embodiment of the inventionmake provision for the creation module GA-M to feature a first creationsubmodule and a second creation submodule, for the first control unit N1to feature the first creation submodule and for the second control unitto feature the second creation submodule. The first control unit N1, thesecond control unit and the control device 30 are connected to oneanother via a data transmission device, in particular wirelessly. Inparticular a data transfer is able to be realized between the modules ofthe first control unit N1, the modules of the second control unit andthe control device 30 via the data transfer device. A control of themedical imaging device 2 by way of the first augmented reality device H1and/or by way of the second augmented reality device H2 is thusoptionally able to be realized. In particular a control command, whichrelates to a movement of the movable component 12, can be output by wayof the first augmented reality device H1 and/or by way of the secondaugmented reality device H2. In particular the first augmented realitydevice H1 and/or the second augmented reality device H2 can feature theestablishment unit 51. Optionally the control device 30 can form a thirdcreation submodule of the creation module GA-M.

The first augmented reality device H1 is worn by a first user U1. Thesecond augmented reality device H2 is worn by a second user U2.Optionally a third augmented reality device H3 is worn by the patient13. With the third augmented reality device H3 the planned part stepsand the results to be expected can be explained by displaying them tothe patient 13 for example in preparation for the intervention and/orduring imaging preparing for an intervention breathing instructions aswell as calming sequences of images can be output.

By way of the first acquisition module R1-M first information can beacquired, which is selected from the group that consists of imageinformation, depth information, coordinate information and combinationsthereof, wherein the first information relates to the medical device 2,IT, MD and/or the medical examination of the patient 13. The firstinformation relates to a spatial area in a field of view of the firstuser U1. The spatial area in particular comprises the intervention areaIA. Without restricting the general thinking of the invention, themedical imaging device 2, the examination device MD and the interventiontool IT are shown by way of example for the medical device in each case.Without restricting the general thinking of the invention, anintervention is shown by way of example for the medical examination. Anintervention tool IT, for example a scalpel, is used for theintervention. In this case it is a matter of the correct position of theintervention tool IT relative to the structure SP of the patient 13. Theintervention is carried out by the first user U1. The second user U2 islocated outside the examination room in which the medical imaging device2 is arranged and in which the intervention is taking place. The system1 makes possible in particular the guidance of the first user U1 by thesecond user U2 in relation to the correct position of the interventiontool IT relative to the structure SP of the patient 13.

FIG. 7 shows a section from the field of view of the second user U2,wherein the field of view of the second user U2 is enhanced with theaugmented reality information AR. A part of the field of view of thefirst user U1 is shown in the field of view of the second user U2 suchthat the second user U2 sees the patient 13, the intervention tool ITand the transfer plate 12 from the perspective of the first user U1. Theaugmented reality information AR is shown in the field of view of thesecond user U2 such that the second user U2 sees the augmented realityinformation AR from the perspective of the first user U1.

The section from the field of view of the second user U2 shown in FIG. 7thus differs from a corresponding section from the field of view of thefirst user U1 in that the patient 13, the patient support device 10, theintervention tool IT are visible as an image and/or as a model for thesecond user U2 and that the arm of the second user U2 is visible as animage and/or as a model for the first user U1.

Based on the first information, examination information, which relatesto a position of an intervention tool IT relative to the structure SP ofthe patient 13, is established. The augmented reality information AR iscreated based on the examination information such that the augmentedreality information AR features a marking EI, which represents themovement of the intervention tool IT to be carried out as the nextmovement, for example in the form of a moving stereoscopic and/orhologram-type representation of the intervention tool IT.

In accordance with the sixth form of embodiment of the invention thesystem 1 features an examination device MD. The augmented realityinformation AR is created based on a medical signal, for example an EKGsignal provided by the examination device MD, such that the augmentedreality information AR features a signal output field DF, wherein themedical signal is shown in the signal-output field DF as a timingcharacteristic.

In preparation for the intervention the patient 13 is identifiedautomatically by way of the first augmented reality device H1, in thatpatient identification information is established based on an image ofthe patient 13, for example with the aid of biometric features. Based onthe patient identification information the augmented reality informationAR is created such that personal patient data and data which relates tothe patient history of the patient 13 will be shown in a patient dataoutput field PI.

Patient position information will be established based on the firstinformation. The augmented reality information AR is created based onthe patient position information such that the position of the augmentedreality information AR is determined relative to the field of view ofthe first user U1 based on the position of the patient 13. In particularthe augmented reality information AR is shown overlaid anatomicallycorrectly on the patient 13.

The creation module GA-M is embodied for creating the augmented realityinformation AR based on the medical imaging dataset. The medical imagingdataset relates to the structure SP of the patient 13. The structure SPcan for example be an organ of the patient 13. The augmented realityinformation AR features a structure marking MI, which marks thestructure SP of the patient 13. The structure marking MI is shownoverlaid anatomically correctly on the structure SP. The structuremarking MI is designed so that a first part area of the structure SP isable to be distinguished from a second part area of the structure SP viathe structure marking MI. In FIG. 7 the first part area of the structureSP is shown as a cross-hatched surface and the second part area of thestructure SP is shown as a white surface. For example the first partarea can involve a part of the organ to be removed and/or the secondpart area a remaining part of the organ. In this way the first user U1and the second user U2 can recognize the first part area of thestructure SP and/or the second part area of the structure SP quickly andexactly.

The augmented reality information AR features the nominal positionmarking TP, which relates to the nominal position of the patient 13 onthe transfer plate 12. The reproducibility between a number of medicalimaging datasets, which are recorded after different part steps of theintervention, is improved.

The augmented reality information AR features the nominal positionmarking TT, which relates to the nominal position of the interventiontool IT. The nominal position marking TT is provided in that the seconduser U2 creates a model of the intervention tool IT by way of the secondaugmented reality device H2 and positions the model of the interventiontool IT in the nominal position of the intervention tool IT. The secondaugmented reality device H2 is embodied to recognize gestures of thesecond user U2 and to create the augmented reality information AR basedon the gestures.

The augmented reality information AR features a region marking SR, whichmarks the region of the transfer plate 12, which is selected inaccordance with the selected protocol for imaging via the medicalimaging device 2. Thus the region marking SR also marks the region ofthe patient 13 that is selected, for an unchanged position of thepatient 13 relative to the transfer plate 12, for the imaging via themedical imaging device 2. The structure marking SP, the nominal positionmarking TP, the nominal position marking TT and the region marking SReach feature a stereoscopic representation and/or a hologram-typerepresentation,

Literature Reference

Ozan Cakmakci and Jannick Rolland: “Head-Worn Displays: A Review”.JOURNAL OF DISPLAY TECHNOLOGY, VOL. 2, NO. 3, September 2006, the entirecontents of which are hereby incorporated herein by reference.

The patent claims of the application are formulation proposals withoutprejudice for obtaining more extensive patent protection. The applicantreserves the right to claim even further combinations of featurespreviously disclosed only in the description and/or drawings.

References back that are used in dependent claims indicate the furtherembodiment of the subject matter of the main claim by way of thefeatures of the respective dependent claim; they should not beunderstood as dispensing with obtaining independent protection of thesubject matter for the combinations of features in the referred-backdependent claims. Furthermore, with regard to interpreting the claims,where a feature is concretized in more specific detail in a subordinateclaim, it should be assumed that such a restriction is not present inthe respective preceding claims.

Since the subject matter of the dependent claims in relation to theprior art on the priority date may form separate and independentinventions, the applicant reserves the right to make them the subjectmatter of independent claims or divisional declarations. They mayfurthermore also contain independent inventions which have aconfiguration that is independent of the subject matters of thepreceding dependent claims.

None of the elements recited in the claims are intended to be ameans-plus-function element within the meaning of 35 U.S.C. § 112(f)unless an element is expressly recited using the phrase “means for” or,in the case of a method claim, using the phrases “operation for” or“step for.”

Example embodiments being thus described, it will be obvious that thesame may be varied in many ways. Such variations are not to be regardedas a departure from the spirit and scope of the present invention, andall such modifications as would be obvious to one skilled in the art areintended to be included within the scope of the following claims.

What is claimed is:
 1. A method for outputting augmented realityinformation to a user, the method comprising: acquiring a medicalimaging dataset relating to an anatomical structure of a patient;acquiring first information, the first information including one or moreof image information, depth information, and coordinate information;creating the augmented reality information by evaluating, via a machinelearning algorithm, the first information and the medical imagingdataset, the augmented reality information including a structuremarking, including a 3-dimensional (3D) image marking the anatomicalstructure of the patient; and outputting the augmented realityinformation in response to the user placing at least one of a hologramof the anatomical structure of the patient and a hologram-typerepresentation of the anatomical structure of the patient in a field ofview of the user, the augmented reality information being combined, inan anatomically correct manner, with the at least one of the hologram ofthe anatomical structure of the patient and the hologram-typerepresentation of the anatomical structure of the patient, and beingperceivable in the field of view of the user.
 2. The method of claim 1,wherein a control command, relating to a movement of a movable componentof a medical device, is output.
 3. The method of claim 1, whereinexamination information, relating to a medical examination of thepatient, is established based on the first information, and wherein theaugmented reality information is created based on the examinationinformation.
 4. The method of claim 1, wherein patient information,relating to the patient, is established based on the first information,and wherein the augmented reality information is created based on thepatient information.
 5. The method of claim 1, wherein the firstinformation relates to at least one of a medical device and a medicalexamination of the patient, wherein nominal position information,relating to at least one of a nominal position of the patient and anominal position of the medical device, is provided, wherein theaugmented reality information is created based on the nominal positioninformation, and wherein the augmented reality information includes anominal position marking, marking the nominal position of at least oneof the patient and the medical device.
 6. The method of claim 1, whereinthe first information relates to at least one of a medical device and amedical examination of the patient, wherein region information, relatingto a region of at least one of the patient and the medical device, isprovided, wherein the augmented reality information is created based onthe region information, and wherein the augmented reality informationincludes a region marking, marking the region of at least one of thepatient and the medical device.
 7. The method of claim 1, furthercomprising: outputting the augmented reality information to beperceivable in a field of view of a second user; and acquiring an inputof the second user, wherein the creating includes creating the augmentedreality information using the input of the second user.
 8. A system foroutputting augmented reality information, relating to at an anatomicalstructure of a patient, to a first user, comprising: at least one memoryand at least one processor, the at least one memory containing computerreadable code that, when executed by the at least one processor,configures the at least one processor to, acquire a medical imagingdataset relating to an anatomical structure of the patient, acquirefirst information, the first information including one or more of imageinformation, depth information, and coordinate information, create theaugmented reality information by evaluating, via a machine learningalgorithm, the first information and the medical imaging dataset, theaugmented reality information including a 3-dimensional (3D) imagemarking, marking the anatomical structure of the patient, and output theaugmented reality information in response to the first user placing atleast one of a hologram of the anatomical structure of the patient and ahologram-type representation of the anatomical structure of the patientin a field of view of the user, the augmented reality information beingcombined, in an anatomically correct manner, with the at least one ofthe hologram of the anatomical structure of the patient and thehologram-type representation of the anatomical structure of the patient,and being perceivable in the field of view of the user.
 9. The system ofclaim 8, further comprising: a medical device, wherein the medicaldevice is embodied to provide the coordinate information.
 10. The systemof claim 9, wherein the medical device includes a movable component andthe system further comprises: a control device configured to, acquire amovement parameter, relating to a movement of the movable component, andestablish the coordinate information based on the movement parameter.11. The system of claim 8, wherein the at least one processor is furtherconfigured to, output the augmented reality information to beperceivable in a field of view of a second user, acquire an input of thesecond user, wherein the at least one processor, configured to createthe augmented reality information, is further configured to create theaugmented reality information using the input of the second user. 12.The system of claim 8, further comprising: a first augmented realitydevice, including a first processor of the at least one processor.
 13. Anon-transitory computer-readable medium, including a computer program,the computer program being loadable into a memory device of a computer,the computer program being configured to execute the method of claim 1when the computer program is executed on the computer.
 14. Anon-transitory memory storing a computer program, the computer programbeing configured to execute the method of claim 1 when the computerprogram is executed on a computer.
 15. The method of claim 1, whereinthe first information relates to at least one of a medical device and amedical examination of the patient, wherein region information, relatingto a region of at least one of the patient and the medical device, isprovided, wherein the augmented reality information is created based onthe region information, and wherein the augmented reality informationfeatures a region marking, which marks the region of at least one of thepatient and the medical device.
 16. The method of claim 1, wherein theacquiring of the medical image dataset relating to an anotomicalstructure of a patient is achieved via a computed tomography (CT) scanof the patient.
 17. The method of claim 1, wherein the acquiring of thefirst information includes acquiring information of the patient via anoptical camera.
 18. The method of claim 16, wherein the acquiring of thefirst information includes acquiring information of the patient via anoptical camera.
 19. The method of claim 1, wherein the first informationrelates to at least one of a medical device and a medical examination ofthe patient.
 20. The method of claim 1, wherein the outputting of theaugmented reality information includes overlaying augmented realityinformation, in the anatomically correct manner, over anatomicalstructure of the patient, the augmented reality information overlaidbeing perceivable in the field of view of the user.
 21. The method ofclaim 1, wherein the augmented reality information includes thestructure marking and a movement marking.
 22. The method of claim 21,wherein the movement marking illustrates a location for an interventiontool in relation to a portion of the anatomical structure.
 23. Themethod of claim 1, wherein the acquiring includes acquiring the firstinformation from the medical imaging dataset.
 24. The method of claim17, wherein the augmented reality information includes the structuremarking and a movement marking.
 25. The method of claim 24, wherein themovement marking illustrates a location for an intervention tool inrelation to a portion of the anatomical structure.
 26. The method ofclaim 17, wherein the acquiring includes acquiring the first informationfrom the medical imaging dataset.
 27. The method of claim 5, wherein theaugmented reality information includes the structure marking, thenominal position marking and a movement marking.
 28. The method of claim6, wherein the augmented reality information includes the structuremarking, the region marking and a movement marking.
 29. The system ofclaim 8, wherein the augmented reality information includes the3-dimensional image marking, and a movement marking.
 30. The system ofclaim 29, wherein the movement marking illustrates a location for anintervention tool in relation to a portion of the anatomical structure.31. The system of claim 8, wherein the at least one processor,configured to acquire the first information, includes acquiring thefirst information from the medical imaging dataset.
 32. The system ofclaim 8, wherein the at least one processor, configured to acquire thefirst information, includes acquiring information of the patient via anoptical camera.
 33. The system of claim 8, wherein the at least oneprocessor, configured to acquire the medical image dataset relating toan anatomical structure of a patient, includes acquiring the medicalimage dataset via a computed tomography (CT) scan of the patient. 34.The system of claim 32, wherein the at least one processor, configuredto acquire the medical image dataset relating to an anotomical structureof a patient, is further configured to acquire the medical image datasetvia a computed tomography (CT) scan of the patient.
 35. The system ofclaim 8, wherein the augmented reality information includes the3-dimensional (3D) image marking, and a movement marking.
 36. The methodof claim 1, further comprising: outputting the augmented realityinformation to be perceivable in a field of view of a second user withthe at least one of a hologram and a hologram-type representation of theanatomical structure of the patient and being perceivable in the fieldof view of the second user; and acquiring an input of the second user,wherein the creating includes creating the augmented reality informationusing the input of the second user.
 37. The system of claim 8, whereinthe at least one processor is further configured to, output theaugmented reality information to be perceivable in a field of view of asecond user with the at least one of a hologram and a hologram-typerepresentation of the anatomical structure of the patient and beingperceivable in the field of view of the second user, acquire an input ofthe second user, wherein the at least one processor, configured tocreate the augmented reality information, is further configured tocreate the augmented reality information using the input of the seconduser.